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ABSTRACT 

A HOW-to-do-it kit for educatorsinterested in 
measuring career path development by utilizing criterion measures 
with domain objectives is presented in the format of a Self-paced, 
career path module, A career path refers to the growth from career 
exploration to career identity, A module is habitually composed of 
four parts: objectives, pretests, learning environments, and 
posttests. Criterion-referenced testing (CRT) is contrasted with 
norm-referenced testing (nrT) , CRT stresses mastery and scoring in 
terms of measurable achievement. Examples are given of CRT items and 
scoring keys, NRT stresses position in the tank order of a group, NRT 
is not recommended as an effective measure of career path progress. 
Diagnostic pretest questions help readers identify needed sections of 
this document while mastery posttests help readers verify correct 
comprehension and application of criterion measures. This document is 
designed for the educator who wants to develop CRT career measures 
but has a limited budget and timetable, (Author/BJG) 
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INTRODUCTION 




There was a time i he mid-1960's when the 
task of the tester was mucn simpler. All that had to 
be done was to thumb through a catalog and order the 
exact number of questionnaires needed. The scoring 
was done by machine. At the time, it seemed like 
there was nothing else that needed to be done to 
come up with local norms. 

Now, ten years later, all that has changed. 
A new term has emerged : criterion-referenced testing. 
CRT has become an abbreviation which is constantly 
contrasted with norm-referenced testing (NRT) . CRT 
is a testing process which utilizes only items directly 
related to prespecified objectives. These objectives 
furnish testing criteria in terms of learner outcomes. 
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This document entitled DEVELOPING CRITEKION 
MEASURES FOR CAREER PATH MODULES is an attempt to 
orgcmi;:e the development of criterion-referenced test 
itetiis. Such test items will serve to measure whether 
or not learners can or cannot accomplish the behavior, 
activity, performance, or skill stated in the criterio 
object i ve . 



Several school districts have spent a large 
amount of money developing CRT systems. The tests 
developed have been transplanted to other systems, 
'n many cases, these transplants did not take because 
of different local conditions. 

It is the contention of this document that 
physically transplanting CRT systems from one district 
to another, from one school to another, from one 
teacher to another' is not an effective innovation device. 

The more effective transplant- would be to trans- 
plant processes and criteria from one CRT developer to 
another. In this way, all CRT performance measures 
used by teachers will possess "local validity" in the 
ser»se that a valid test ite.ti for a highly specific 
objective may lack validity for a highly specific local 
objective just a little bit different. 
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This process of developing local CRT performance 

measures can be sunrmar i zee! in two stages : 

STAGE ONE wherein teachers make 
a first attempt ot 
ana I yz i ng, organ i z i ng, 
and packaging a combin- 
ation of existing and 
new matcf* i a I s 

STAGE TWC wherein teachers make 
a more formal attempt 
to evaluate, articulate, 
and systematize a wide 
variety of materials 
from colleagues and 
prof ess i ona I peers . 



Each section of this document is intended to 
clarify the process of developing CRT performance 
measures for CAREER PATH MODULES. A career path refers 
to the growth from career exploration to career identity. 
A module is hobitually composed of four parts : objectives 
pretests, learning environments, and posttests. 

This document will be in the format of a module. 
OBJECTIVES will be stated. Diagnostic PRETESI test items 
will be available to the readers. Specific LEARNING 
ENVIRONMENTS will discuss STAGE ONE and STAGE TWO. A 
mastery type POSTTEST will enable readers to confirm 
understanding, application, and appreciation of this 
process of developing per f orrrance measures. 



6 

Iv 



IN'TnOP!ICTIO\' 10 N'ODIiLCS 



This document is in the format of a MODULE. 
As all modules, it contains four essential elements : 
A I ist of OBJECTIVES 
Self -Assessment PRETESTS 
Alternative LEARNING ENVIRONMENTS 
Mastery POSTTESTS 

The OBJECTIVES give an idea of what to expect 
in the. module. Objectives are unambiguous statements 
of intentions and expectations. Objectives as such can 
include knowledge objectives (KG), performance objectives 
(PC), attitude objectives (AO), basic research, proven 
strategies, valuable attitudes, significant opinio.-.s, 
basic skills, and relevancy analyses. , 

The PRETESTS allow readers to preassess module 
Information, skills, and attitudes already possessed. 
The results of the pretest indicate where to begin the ' 
module and where to go for help appropriate to the reader's 
level of previous accomplishments. Pretests can be oral, 
written, performance, manual, or interaction-based. 



ERIC 



p 



0 

ERIC 



The LEARNING ENVIRONMENTS can range from instructor- 
based to learner-center activities, demonstrations, methods, 
media, or strategics. The selection of the proper learning 
environment is based upon previous success, instructional 
experiences available, training background, expertise in 
the module objectives, favorite style or mode of learning, 
and reasonable time limit expectations. 

The POS.TEST is an attempt to measure the learner's 
growth. The posttest measures how well the original 
module objectives have been met. Sometimes, the posttest 
may repeat items from the pretest. However, in the case 
of the posttest, the objective is measurement of mastery. 
Diagnosis is the objective of the pretest. Documentation 
of competency is the corresponding objective of the posttest. 

Posttests can take many forms : teacher-constructed 
evaluotion, performance-based task completion, learner 
self-assessment, contractual agreements, standardized 
instruments, rating scales, subjective impressions, 
minimum essentials performance, anecdotal records, or 
any combination appropriate to a given learner at a given 
time in the learning process. 

This module contains all four ingredients listed 
above : objectives, pretest, learning envt ronlncnts, 
and posttest. The next page explains how to use this module. 
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HOW TO USL TIMS MODULE 



The following steps can simplify the utilization 
of this module and maximize the effectiveness to the 
reader. 



STEP ONE - Examine coreful ly the objectives 

If the objectives are already mastered, 
verify this cloimcd accomplishment by 
TAKING THE PRETEST. 

If the objectives are of such a nature 
as to be as yet unottaincd, begin by 
WORKING THROUGH THE LEARNING ENVIRONMENTS. 



STEP TWO - Take the pretest 

If the pretest is successfully passed 
according to the suggested answer key, 
proceed to verification of mastery by 
TAKING THE POSTTEST. 

If the prt,t'est is not successfully passed, 
proceed to an appropriate learning 
activity by WORKING THROUGH THE 
LEARNING ENVIRONMENTS. 



STEP THREE - Work through the learning environments 

Look over the alternatives available. 

Begin with a learning environment that 
•seems to promise interest and success. 

If it is felt that some of the objectives 
can be met at this time, diagnose this 
situation by TAKING THE PRETEST. 

If It is felt that most of the objectives 
are now mastered, verify this impression 
by TAKING THE POSTTEST. 
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STEP FOUR - Take the posttest 

If the posttest is successfully passed, 
go on to an appropriate next, step, 
for example, another module or on 
alternate learning procedure. 

If the posttest is not successfully 

passed, analyze the missing competencies 

by REEXAMINING THE OBJECTIVES, or 

by RETAKING THE PRETEST, cr 

by AGAIN WORKING THROUGH THE LEARNING 

ENVIRONMENTS provided in the module. 



The above four steps can simplify the process 
of getting used to I earn i ng- w i th modules as vehicles 
of self-instruction^ self-assessment, and se I f-determ i n- 
at i on of goa I s . 

In order to keep track of the needed module 
ingredient, i.t is suggested that the reader refer to 
the TABLE OF CONTENTS in order to find the beginning 
or first page for each of the following : 

OBJECTIVES 

PRETEST 

' LEARNING ENVIRONMENTS 

POSTTEST 

These four components can wel I become a method 
of learning on one's own in a variety of alternatives. 
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Expert users of modules have found an uncomplicated 
and direct way to analyze, organize, and structur^e the 
many parts of a typical n.odule. 

The first step is to recognize that all modules 
are composed of OBJECTIVES, EVALUATIONS (from which both 
PRETESTS and POSTTESIS can be developed), and RESOURCES. 

The second st€*p is to subdivide objectives, 
evoluutions., opr» resources into KNOWLEDGE, PEKTCRMANCE, 
and ATTITUDE cotnpononts . 

The third step is to specify three types of 900 1 s : 
KNOWLEDGE OBJECTIVES or KO, 
PEi;rORN'.AUCE OBJECTIVES or PO, and 
ATTITUDE OBJECTIVES or AO. 

The fiftli step is to specify three types of tests : 
KNOWLEDGE EVALUATION or KE, 
PERFORMANCE EVALUATION or PE, and 
ATTITUDE EVALUATION or AE. 

The sixth step is to specify three" types of tools? : 
KNOWIEDGE RESOURCES or KR, 
PERFORMANCE RESOURCES or PR, and 
ATTITUDE RESOURCES or AR. 

This gives en easy to 

renieniL^er 3x3 matrix : . KO PO AO 

KE Fi; AE 
KR PR AR 
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OBJECTIVES FOR 
DEVELOPING CRITERION MEASURES FOR CAREER PATH MODULES 



As a result of studying this document, the reader will be able to: 

1. Translate local objectives for career education into 
measurable criterion objectives. 

2. Develop criterion-references testing measures that provide 
pretest and posttest data upon which to determine gains 
scor-es in career education. 

3. Quantify measures of the effectiveness of various career 
path programs. 

4. Plan career education upon a framework of specific data 
and information. 

5. Break out of the built-in limitations of norm referenced 
testing. 

6. Incorporate a large variety of career path training modules 
available from other schools, agencies, institutions, and 
organizations sharing similar career education objectives. 

One fundamental observation must be made about these objectives. 
The traditional system of marking, examing, grading, credit granting, 
and advancement is open to distortions as far as career path growth is 
concerned. Good teaching, as well as measurable learning, presumes an 
objective standard. An objective standard is something against which 
progress can be measured. This progress is called a gains score. 

Students in career education imagine there is some sort of standard 
towards which the students are working. This standard is seldom spelled 
out in detail either to parents or to students or to teachers. Employers 
assxmie that the learner's credentials denote achievement of some standard 
of career education. Compared with today's standards, the origin of a foot 
as anything from a size 5 to a size 15 is remarkably precise. 
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This document is intende^^ to help teachers measure objectives that 
can be measured, not by comparing one learner against another, but by 
comparing each learner against chosen or imposed objectives. More and more, 
teachers are beginning to use such criterion objective . .i asis of 
evaluation.. In certain subjects, criterion measures are obvious. In the 
area of career path growth, the criterion measures must be made more obvious, 
and understandable to the average learner. 

Career path criterion measures offer several promises: 

A. Accurate self-evaluation must be sponsored. * 

B. All learning objectives should be appraised. 

C. Teaching and learning should be organized around objactives. 

D. Records should be used for decision making. 

E. Reliable norms for decision making should be based upon criterion 
values. 

It is obvious that one principle runs through all attempts to measure 
career path gains: Learners are judged on more than what they know. This 
refers to basic skills and attitudes essential to the learning process. 

Criterion measures fcrce teachers to layout expectations in terms learners 
can understand. This fosters mutual cooperative and directly leads to better 
gains scores. 
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jjj for 

DEVELOPING CRITERION MEASURES FOR CAREER PATH MODULES 



DIRECTIONS : Indicate ci I I cinsuers chosen on the 
ANSWER SHEET provided. 



Verify all answers selected by 

consulting the ANNOTATED SUGGEST EU ANSWER KEY, 



TRUE FALSE I . 

TRUE FALSE 2. 

TRUE FALSE 3. 

TRUE FALSE 4. 



CRT abbreviates criterion-referenced, 
test i ng . 

NRT abbreviates norni-i*ef ernced testing. 

CRT and NRT' are essentially synonymous. 

CRT per f or'njdnc e measures utilize only 
rest ftems directly related to pre- 
specified objectives. 



TRUE FALSE 5. The object ives used to specif> criteria 
for CRT perforniance measures are in 
terms of learner outcomes, performances, 
skills, attitudes, and growth. 

TRUE FALSE 6. CRT per'forniance measures are used to 
determine whether or not students 
can perform es expected in the 
specific criterion objectives under 
. eva I uat ion, ^ 



TRUE FALSE 
TRUE FALSE 



TRUE FALSE 
TRUE FALSE 



7. NRT evaluative measures ore intended to 
provide group norf»,s. 

8. NRT EVALUATICM MEASURES use group 
norms which show the standing of one 
individual in relation to otFers in 
the group being tested. 

9. Percentiles are used in CRT and not at 
all in NRT. 

10. Usually, perceniiles must be interpreted 
in terms of prespecified objectives. 
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DIRECTIONS : Some multiple cho ice have more than one answer. 



11. With a good set of NRT evaluative measures, a 
teacher is abl e to 

A. Rank individual learners in the group with 
greater accuracy 

B. Diagnose specific learning difficulties in 
terms of objectives to be attained 

C. Show the relative standing of one individual 
testee in relation to others taking the 
same NRT evaluative measure 

D. Measure mastery of prespecified learner outcomes 

E. Zero in on prespecified standards of mastery 

12. With a good set of CRT performance measures, a 
teacher is cible to 

A. Develop group percentiles 

B. Mark on the curve 

C. Establish undisputable class ranks 

D. Measure mastery, partial mastery, or lack of 
mastery of prespecified criterion objectives 

E. Use stcnines in scoring exams 

13- When a learner fails a CRT performance measure, 
he is to I d 

A. Seventy per cent of the class did better than you 

B. It's your fault you failed 

C. You failed to achieve the desired mastery outcome 
AT THIS TIME 

D. You never will get another chance to try again 

E. Better luck next time since it's all a matter 

of luck rather than lack of competency on your part 

14- In a CRT performance measure environment type of 
classroom, the teacher 

A. Must give every student an " 

B. Must compute grades on the curve and on the percentile 
system and then give each learner the highest grade 

C. Usually explains the criterion objectives before 
testing for mastery 

D- Must give out at least fifty per cent of the actual 
test questions to be used before the test date 

E. Maintains no record of how many objectives were 
accomplished to date by each learner but relies 
instead on group norms 
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ANNOTATED SUGGESTED ANSWER KEY FOR 
PRETEST FOR 

DEVELOPING CRITERION MEASURES FOR CAREER PATH MODULES 



ANSWERS 


ANNOTATION 


1. 


True 


CRT 


2. 


True 


NRT 


3. 


False 


Contrasting approaches 


4. 


True 


Prespecified objectives 


5. 


True 


Measurcable outcomes 


6. 


True 


Visi ble outcomes 


7. 


True 


Group norms 


8. 


True 


Relative rank 


9. 


False 


NRT uses percentiles 


10. 


False 


Percentiles are interpretated in terms of groups 


11. 


A, C 


NRT ranks group members 


12. 


D 


CRT measures mastery 


13. 


C 


CRT always provides another chance 


14. 


C 


Individual norms prevail 
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LEARNING ENVIRONMENTS for 



DEVELOPING CRITERION MEASUREMENTS 

STAGE ONE - The First Attempt 

The following pages present an abbreviated example 
of developing criterion measurements. This example is 
abbreviated because several of the brainwashing steps 
which led up to the final product have been eliminated. 
This elimination was done to focus more directly on the 
type of product that can be expected from classroom 
teachers applying these techniques of criterion measure 
deve I oprnent . 

The following page entitled PRETEST DATA : COUNTABLES 
is intended to sum up the data needed by the school personnel 
in order to measure the impact of a special program. Thus, 
it was deemed necessary to specify four factors : 



FACTOR A 



Diploma Potential 



FACTOR B 



Testab I es 



FACTOR C 



Products 



FACTOR 5 



Curr i cu I um 



Inspection of the following page can reveal the 
operational definition of each factor being measured. 
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PRETEST DATA 



COIINTABLES 

D i p I oma Potent i a I 

Cateflory Per Cent Cumulative Per Cent 

Grade School Graduate 15 | 00 

High School Graduate 45 85 

Community College Graduate 20 40 

Four Year College Graduate 20 20 

Testab I es 

Pretest and posttest scores on the eight compc nents of Career Educatio 
Pretest and posttest scores on five career clusters. 

Products 

Year I - 5 career clusters in the form of learner-paced modules 

Year 2 - ^ more career clusters 

Year 3 - 5 add i t.i pnaj , career , cl usters 



Curr i cu I um 

The curriculum will be SEQUENTIAL in the sense that f i ve 
different clusters will be studied in each of three years. 

The curriculum will be CYCLICAL in the sense that the eight 

career education components will be studied in more depth each year. 
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A SAMPLE OF COUNTABLE PRETEST DATA 

FACTOR B (TESTABLES) from the preceding page refers 
to pretests on the eight components of Career Education. 

One of the eight components of Career Education is 
commonly referred to as the development that occurs as the 
learner grows from SELF-AWARENESS to SELF-IDENTITY. 

There are many possible ways to measure SELF-AWARENESS 
growth. One way is to give the PRETEST found on the following 
page at the beginning of the school year and to give a similar 
POSTTEST at the cl ose of the school year. The difference in 
learner scores indicates growth if the posttest is higher in 
desired traits. 

The following page is the PRETEST measuring growth 
from SELF-AWARENESS to SELF-IDENTITY. This test was 
developed by junior high school educators. This page is 
given to students. 

The next page entitled ANSWER SHEET is also given 
to learners. Since students do not write on the PRETEST page^ 
all responses are put on this answer sheet. 

The following page after the answer sheet is 
entitled SCORING KEY. This page is for teachers only. 
The scoring key explains how to interpret and analyze 
the answer sheet of each student. 
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QUESTIONNAIRE 
DIRECTIONS: Place answers on special ANSWER SHEETS provided. 

PRETEST 

from SELF-AWARExVESS to SELF-IDENTITY 



1. Which is most important to you? 

A. Money 

B. Good paying job 

C. Community status 

D. Work 1 like 



2. Which is 2nd most important? 



3. Wliich is 3rd most important? 

4. Which is least important? 



5. What activity would you most enjoy participating in? 

A« Going on a tour of Washington, D.C. 

B. Watching a baseball game 

C. Working in the garden 

D. Managing a baseball team 

E. Organizing a local ciub 

6. Which is 2nd most important? 



7. Which is 3rd most important? 

8. Which is least important? 

9. Describe what you think o£ yourself in each of the following categories: 

A. Appearance: 

B. Personality: 

C. Interest: 

D. Abilities: 



10. How would you like to be different in each category? 
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ANSWER SHEET 

from SELF-AWARENESS to SELF-IDENTITY 

1. A B C D 

2. A B C D 

3. A B C D 

4. A B C D 

5. A B C D E 

6. A B C D E 

7. A B C D E 

8. A B C D E 

9. A, Appearance: 



B, Personality: 



C. Interest: 



D. Abilities: 



10. A. 



B. 
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SCORING KEY 

PRETEST from SELF-AWARENESS to SELF-IDENTITY 



1. A little more introspection is evident if the response 

2. changes from money as high to money as low. 
3. 

4. 



5. ^ A little more realism is evident if Hi-pre A, B changes 

6. to Low-post A,B and if Low-pre C,D,E changes to Hi-post C,D,E, 
7. 

8. 



9. 1 point for SPECIFIC 
1 point for MEASURABLE 



10. 1 point for POSITIVE 

1 point for POSSIBLE (REALISTIC) 
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LOOKING BACK OVER STAGE ONE 



Stage One can be summed up in the fo I lowing 
sequence : 



1st — Several educators tried to 
pinpoint baseline data. 

This resulted in 

PRETEST DATA : COUNTABLES. 



2nd — These educators tried to 
gather PRETEST DATA on 
present student levels 
of GROWTH FROM SELF-AWARENESS 
TO SELF- IDENTITY 

This resulted in a PRETEST : 
from SELF-AWARENESS 
to SELF-IDENTITY. 



3rd — This PRETEST was operat i ona I i zed 
in three parts : 

I: A QUESTIONNAIRE to be 
given to each student. 

2: An ANSWER SHEET upon 
which students are 
to write answers to 
the quest i onna i re, 

3: A SCORING KEY with which 
teachers and educators 
• are to score and interpret 
the responses found on 
answer sheets filled out 
by students. 

The QUESTIONNAIRE, the ANSWER SHEET, and the SCORING 
KEY referred to appear on the preceding three pages. 
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STAGE TWO - A Forrnalized Attempt 



Further reflection upon the materials developed in 
stage one led to the recognition of the need for a more 
systematic and formalized approach to the problem of providing 
an adequate PRETEST. A more adequate pretest was operationally 
defined as an evaluation instrument which uould lend itself 
to testing (a pretest in the first weeks of the school year) 
and to retesting (a posttest in the last weeks of the school 
year) in order to obtain a GAINS SCORE. 

The GAINS SCORE was defined as the difference between 
the pretest and the posttest taken by each individual. The 
average of the individual gains scores (both positive and 
negative) was defined as the GROUP GAINS SCORE. 

This PRETEST was to be a criterion-referenced eval- 
uation instrument containing only items directly related to 
previously stated objectives. Each of these prestated 
objectives was to be a student outcome. In the example found 
in stage one, the student outcome was to "grow from self- 
awareness to self-identity." In stage two, the student outcome 
is to "grow from educational awareness to educational identity," 
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In stage one, the format of the developed product 
was as fol lows : 

I: QUESTIONNAIRE 
2: ANSWER SHEET 

3: SCORING KEY (from which prespecified 
objectives could be inferred). 

It was found that going backward from a test item 
to an implied objective was possible. However, such a 
path seemed backward since it would be more direct to 
state an objective and then to seek out or develop an 
appropriate evaluation or test item. 

In addition, it was felt that greater thoroughness 
would result from subdiving a concept such as EDUCATIONAL 
AWARENESS into component parts. This would avoid testing 
only superficial and external aspects of this important 
starting point for EDUCATIONAL IDENTITY. 

In this way, the format of stage two emergetJ as 
f o I I ows : 

I: CRITERION OBJECTIVES 
2: TEST ITEMS 
3: ANSWER SHEET 
4: SCORING KEY. 
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In this example^ simplicity has been stressed by 
giving only one test item per objective. 

For example^ objective I is measured by test item 

1 on the following page. If it had been desired to give 
two more test items with which to measure objective 1^ 
one convenient way to keep track of this correlation 
would be to number these test items IB and IC. 

Similarly^ objective 2 is measured by test item 

2 on the following page. 



To see how wel I thys reader understands thfi f'ol lowing 
pages, an example is provided. Try to determine the PRETEST score 
of a student vvhose answer sheet looks like this : 

I.0BCDE 4. ABC®E 

2. A B (c) D E 

3. A ® C D E 

Try to determine the POSTTEST score of a student 
whose answer sheet looks like this : 
I . A B C (d) E 

2. A B C (d) E 

3. A B C (d) E 

4. A B C D (e) 

As a final exercis*., try to determine the GAINS 
SCORE FOR THIG INDIVIDUAL STUDENT. Don't look at the 
answers given after the scoring key until after trying to 
figure it out beforehand. 
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EDUCATIONAL AWARENESS 
CRITERION OBJECTIVES 

Each learner will: 

1. Specify a personal reason for going to school* 

2, Back up this personal reason with data, 

3. Plan post-secondary job or college in terms of a career, 

4, Identify at least one educational goal to accomplished 
in junior high school. 
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EDUCATIONAL AWARENESS 



TEST ITB^IS 



Directions: Place all answers on ANSWER SHEET. 



1. What is the most important reason why you go to school? 

A. To learn things to prepare me better for my future. 

B. 1 have no place else to go. 

C. 1 am forced to. 

D. 1 like school. 

E. 1 don't know. 

2. How will you decide which high school program to attend? 

A. I will go where my friends go. 

B. 1 will seek advice from parents, counselors, or teachers 

C. 1 have no idea. 

D. 1 will seek a high school where I can develop my interes 

and abilities. 

E. 1 will go to the high school where the work is easiest. 

3. How will you decide what to do after high school? 

A. 1»11 have to find- out if the job I want requires more 

education or training, 

B. 1 won't think about it till after I graduate. 

C. I'll do what my parents want me to do. 

D. It depends upon my grades and interests. 
E» I have no idea. 

4. What's your most important goal at school this year? 

A. Getting good grade s^ 

B. Staying out of trouble. 

C. Learning more about my favorite subject. 

D. Passing with as little homework as possible. 

E. I don't know. 
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EDUCATIONAL AWARENESS 



ANSWER SHEET 



ERJC 



1. A B C D E 

2. A B C D E 

3. A B C D E 

4. A B C D E 
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EDUCATIOML AWARENESS 



SCORING KEY 



!• Positive: A, D 
Neutral : E 
Negative: B, C 



2. Positive: D 
Neutral : C 
Negative: A| E 

3. Positive: A, D 
Neutral : E 
Negative: B, C 



4. Positive: A, B 
Neutral : E 
Negative; C, D 



DIRECTIONS: 

Only one choice may be scored per test iteiA; 
Answer sheets with more than one choice circled by students 
are to be vo i d^-cj- 

Give one point (+l) for each positive response. 

Give minus one point (-1) for each negative response. 

Give zero points (O) for each neutral response. 

Add up the total for all four questions to obtain 

the INDIVIDUAL SCORE which may range between (-4) and (+4). 

For example^ (+l) plus (O) plus (-1) plus (-1) 

equals a TOTAL INDIVIDUAL SCORE of (-1). 

Similarly, (+l) plus (4|) plus (+1) plus (O) 

equals a TOTAL INDIVIDUAL SCORE of (+3). 
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ANSWERS FOR THE PRECEDING EXAMPLE 

The preceding example which analyzed the last 
four pages can be answered as follows: First, the 
question number for each test item (ITEM NUMBER), second, 
the student's response (STUDENT RESPONSE), third, the 
scork'ing key weight (SCORING WEIGHT), and finally the 
TOTAL INDIVIDUAL SCORE. 

PRETEST 



ITEM NUMBER STUDENT RESPONSE SCORING WEIGHT 

' A +1 

2 C 0 

3 B -I 

4 D -I 
TOTAL INDIVIDUAL SCORE (-|) 

POSTTEST ". ' * 

1 D +1 

2d +1 
3d +1 
4 E 0 

TOTAL INDIVIDUAL SCORE (+3) 
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For this student, the PRETEST is (-1) and 
the POSTTEST is (+3). 

The formula for the INDIVIDUAL GAINS SCORE is 
(POSTTEST) minus (PRETEST). 

Recalling the laws of addition and subtraction for 
sign'-d numbers, it can be demonstrated that 
(+3) minus (-1) equa I s (+4). 

This means that the INDIVIDUAL GAINS SCORE 
for this individual is (+4). 

The plus sign (+) indicates a positive gains 
score. A negative sign (-) would indicate a negative 
gains score. A negative gains score means that the 
learner's posttest was lower than the learner's pretest. 

So much for the mathematics of scoring the four 
test items used in this example. 

The next few pages will delve into the underlying 
principles that led up to the test items and the scoring key. 
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PHASE ONE : Teacher-ITeve I oped Questions 



After stating a criterion objective, each 
teacher developed a correlated multiple choice test 
item with five alternatives. 



FOR EXAMPLE : 
a. An objective was stated : 



THE LEARNER WILL 

SPECIFY A PERSONAL REASON 

FO'^ GOING TO SCHOOL 



b. A question was developed : 



WHAT IS THE MOST IMPORTANT 
REASON WHY YOU GO TO SCHOOL? 

c. Alternatives were provided : 

A. To learn things to prepare me better 
for my future 

0. I I i ke schoo I 

E. I don't know 

B. I have no place else to go 

C. I am forced to 

d. Alternatives were weighted to provide 2 positive 
weightings of (+l), one neutral weighting of (O), 
and two negative weightings of (-1). 

In the opinion of the teachers doing the 
weightings, A and D were (+l) 

E was (0), and 

B and C were (- I ) . 



This is the process used to construct the sample 

of four test items discussed in this stage two. It i j5 

appropriate at this point to ask a few questions about 
the validity of this approach* 
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This approach permits teachers to recall typical 
responses indicative of positive, neutral, or negative 
individuals. Usually, subjecting such suggested responses 
to the professional opinion of the teachers concerned is 
enough for local purposes. 

For larger samples, a statistical sampling might 
help come up with alternative responses which have the 
fol lowing characteristics : 

RESPONSES COMING FROM A STRATIFIED SAMPLE will 

a. be closer to actual language of 
the population being tested 

b. reflect more clearly the actual 
positive and negative concerns 
of the target population 

c. be less contaminated by teacher 
or tester unconscious bias 

d. give a flavor of reality to 
the teacher-de ve I oped quest i one 
by incorporating I earncr~ed I ted 

• ^alternatives to tiie questions 
under i nvest i gat i on 

e • prov i de a I arger samp I e of 
learner response from which 
testers can construct bias- 
free a I ternat i ves . 

Thus, the next phese is learnei — edited questions. 
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PHASE TWO : Learner-Edited Questions 

Phase two can be inaugurated very siniply- 
The teacher develops an open-ended questionnaire. 
In this case^ the questions would be : 

1. WHAT IS THE MOST IMPORTANT REASON 
WHY ^01! GO TO SCHOOL? 

2. HOW WILL YOU DECIDE WHICH HIGH 
SCHOOL PROGRAM TO ATTEND? 



3. HOW WILL YOU DECIDE WHAT TO 
DO AFTER HIGH SCHOOL? 



4. WHAT'S YOUR MOST IMPORTANT 
GOAL AT SCHOOL THIS YEAR? 

The teacher tests a sample of learners from 
each of several possible strata. 
For examp I 

5 learners could be tested from each of 
the top two groups 

10 learners could be tested from the 
m i dd I e group 

5 learners could be tested from each of 
the bottom two groups 

This sampling is for the teacher with five groups. 

The teacher categorizes the sample responses to 
the open-ended questions as either positive, neutral, 
negative in light of the criterion objective for each 
test item. 



35 



The teacher edits the alternatives in the multiple 
choice questions in light of the open-ended responses. 
This permits a refined testing instrument to be used to 
gather both pretest and posttest data. 



This detailed execution of phase two is not 
presented in detail. 

It is left as an exercise to the interested 

reader. 



A FEW GUIDELINES ARE IN ORDER : 

1. The above examples, both in stage one and in stage 
two are presented to provoke thought rather than to 
prescribe THE ONLY WAY TO DO IT. 

2. For all practical purposes, local validity means 
satisfying all local participants as to the worthwh i I eness 
of any pretest or posttest used. 

3. For statistical validity, the pretest and posttest used 
to determine must be identical or similar to provide a val 

V 
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comparison of the effect of a program on the test 
scores of selected learners. 

FOR EXAMPLE, if the PRETEST used were the one developed 
in phase one, it would not be appropriate to use the 
P0STTES1 developed in phase two^ even though the stems 
of each question were identical. Changing, and improvifig, 
the alternatives changes the nature of the testing 
i nstrument . 

The only exception to the above would be statistical 
proof that a pretest such as the one developed in phase 
one were statistically equivalent to a posttest such as 
the one developed in phase two- 

|f there were a stage three, it would be entitled : 
DO IT YOURSELF,* YOUR OWN WAY, WITH LOCAL PERSONNEL. 

After you have had a chance to work out phase two 

on your own, look at the following pages. They reveal what 

one teacher did in applying phase one and phase two in 

his own school. The patterns used was : 

CRITERION OBJECTIVES 
TEST ITEMS 
ANSWER SHEET 
SCORING iXcY 

Perusal of the following pages will reveal local adaptations. 
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SELF-AWARENESS 
CRITERION OBJECTIVE 

The learner will be able to; 

1. Identify at least two areas of interest in school and out of school 

2. Specify at least two academic and two non-academic 
strengths or abilities 

3. Identify at Itiast two areas where improvement is 
desired 
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SELF-AWARENESS 



TEST ITEMS 

I can describe my interests in school in the following way: 

A. I'm not especially interested in anything at school 

B. I can list two things I'm iuter^sted in at school. 

They are: 

1st 

2nd 

C. I can't answer this question 

I can describe my interests outside of school In. the following way 

A. I'm not esnecially interested in anything outside of school 

B. I can list two things I'm interested in outside of school. 

They are: 

1st 

2nd - ' 

C. I can't answer this question 

I can describe my abilities in school in the following way: 

A. I'm not especially good at anything in school 

B. I can list the following two things I'm good at in school. 

They are: 

1st ^ 

2nd 

C. I can't answer this question 
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4. I can describe my abilities outside of school in the following way: 

A. I'm not especially good at anything outside of school 

B. I can list the following two things I'm good at outside of school. 

They are: 

1st 

2nd 

C. I can't answer this question 



5. "Where do you feel you desire to improve? 

A. There's nothing I'd like to be better at. 

B. I can list two things I'd like to be better at. 

They are: 

1st 

2nd 
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SELF-AWARENESS 

ANSWER SHEET 

(if you choose B, fill in below) 

1st 

2nd 

(if you choose B, fill in below) 

1st 

2nd 

(if you choose B, fill in belcw) 

1st 

2nd 

(if you choose B, fill in below) 

1st 

2nd 

(if you choose B, fill in below) 

1st 

2nd 
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SELF-AWAP^.NESS 
SCORING KEY 

FOR ALL QUESTIONS: 

Positive: B 
Neutral: C 
Negative: A 
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LOOKING BACK OVER STAGE TWO 

The following page entitled AN EXAMPLE OF 
DEVELOPING CRITERION MEASURES outlines how ta correlate 
GOALS and RESOURCES. 

Phase one, and the role of teachers in 
developing criterion measures, is outlined in detail. 
Phase one enables a teacher to go through the entire 
process relying upon the best of recalled student 
experiences and responses. 

Phase two, and the role of learners in 
editing questions and alternative responses, is 
similarly outlined in detail. Phase two enables 
educators to draw upon both statistical sampling and 
available stu_dent samples in order to document the 
criterion measures locally developed. 

The message is obvious : 
STUDENTS VARY FROM SCHOOL TO SCHOOL. 

What worked in one school might not work in another. 
TRANSPORTING PROCESSES SUCH AS PHASE ONE AND PHASE 
TWO WILL LEAD TO LESS ERRORS THAN TRANSPORTING TEST 
ITEMS WHOLESALE FROM ONE PROGRAM TO ANOTHER. 

The following page summirizes these processes 
which can be transported safely from one school to another. 

' 43 

ERIC 



An Example of Developing 
CRITERION MEASURES 



® GOALS 



OBJECTIVE ONE is to: 



Develop criterion- referenced tests to help evaluate growth of 
individual children. 



(2) RESOURCES 



PHASE ONE ; TEACHER-DEVELOPED QUESTIONS 



One process to attain this objective can include four steps: 
Step 1 : State a criterion objective 

Step 2 : Develop a correlated multiple choice test item 
with five alternatives 

Step 3 : Plan a student ANSWER KEY 

Step A : Specify the SCORING KEY with 2 positive, 1 neutral, 
and 2 negative responses identified 

" phase TWO : LEARNER-.EDITED QUESTIONS 



Each of the test items can also be asked in an open-ended fashion to a 
sample of learners. 

Step 1 : Develop open-ended questionnaire 

Step 2 : Test a sample of 5 learners from each of the 
five Strata 

Step 3 : Categorize sample open-ended responses as either 
positive, neutral, or negative 

Step 4 : Edit the alternatives in the multiple choice 

questions in light of the open-ended responses 
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POSTTEST for 

DEVELOPING CRITERION MEASURES FOR CAREER PATH MODULES 

DIRECTIONS : Indicate all answers chosen on the- 
ANSWER SHEET PROVIDED. 

Verify all answers selected by consulting 
the ANNOTATED SUGGESTED ANSWER KEY. 

I • A CRT performance measure 

A. Always provides percentiles 

B. Always provides group norms 

C. Sometimes provides criterion objectives 

D. Always provides criterion objectives 

E. Sometimes ignores desired student outcomes 

2. A NRT evaluative measure 

A. Is based upon a prespecified learner objective 

B. Is impossible without a complete behavioral objective 

C. Is helpful in determing group percentile comparisons 

D. Utilizes only test items directly related to 
previously stated learner objectives 

E. Provides only a pass or fail type of scoring 

3. A student who is bragging about his NRT evaluation 
results will be inclined to say 

A. I made the third highest grade in a class of 
th i rty-f i ve students 

B. I arhived only the first three objectives of 
the seven required for complete mastery 

C. I still have only one more objective out 
of nineteen to achieve 

D. I'm good in math graphing but the math 
formulas give me a lot of trouble 

E. I've accomplished most of the career awareness 
subobject i ves but I've still got a number of 
career identity objectives to attain 

4. A high rank in NRT evaluation is insignificant if 

A. Everyone got a high rank 

B. The highc>st ranking individual hasn't achieved much 

C. All clasfi members were able to demonstrate competency 

D. Ninety per cent of the group are over-achievers 

E. Five per cent of the group are under-ach i evers 
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TRUE FALSE 

TRUE FALSE 

TRUE FALSE 
TRUE FALSE 



5. An INDIVIDUAL PROFILE on self-awareness 
or on career awareness summarizes the 
test results of one individual. 

6. A GROUP PROFILE on self-awareness or on 
career awareness summarizes the test results 
of several learners arranged in a group. 

7. The PRETEST RESPONSE of a learner is in 
terms of a particular response chosen. 

8. For all PRETEST RESPONSES, alternative 
A would have the same meaning on all 
quest i ons. 



TRUE FALSE 9. For all POSTTEST RESPONSES, alternative 

A would have the same meaning on all 
quest i ons . 



TRUE FALSE 10. Both the PRETEST RESPONSE and the POSTTEST 

RESPONSE must be translated into the approp- 
riate code, that is, (+l) or (O) or (-1) 
in order to "classify the meaning of the 
response chosen by each individual. 



TRUE FALSE I I 



It is conceivable that a 
of B on question 1 could 
PRETEST RESPONSE of B on 
be (-1). 



PRETEST RESPONSE 
be (+l) wh ilea 
question 2 could 



way, a PRECODE 6f (+1) on 



TRUE FALSE 12. in the same 

question 3 is treated differently than 
a PRECODE of (+l) on question 4. 



TRUE FALSE 13. 



A learner with a hOSTCODE OF (O) and 
PRECODE of (-1) on the same question 
achieved a gains score of (+1). 



a 

has 



DIRECTIONS 



Some of the following multiple choice questions 
may have more than one required correct 
a I ternat i ve . 



14. The PRECODE was (O) and the POSTCODE was 
the GAINS SCORE for this question was 



(O), therefore 



A. 
B. 
C. 



zero 



D. (+2) 

E. Indicative of growth 

F. Indicative of decline 



46 



15. The POSTCODE was (O) and the PRECODE was (-1), 
therefore the GAINS SCORE was 

A. zero D. (+2) 

^' E. Indicative of growth 

^' F. Indicative of decline 



16. The POSTCODE was (-1) and 
therefore the GAINS SCORE 

A. zero 



the PRECODE was (O), 
WAS 

D. (+2) 

E. Indicative of growth 

F. Indicative of decline 



17. A GAINS SCORE of (-1) 

A. Means nothing has happened 

B. Is indicative of decline* 

C. Is indicative of growth 

D. Meons the PRETEST was higher than the POSTTEST 

E. Means the POSTTEST was lower than the PRETEST 



18. A GAINS SCORE of (O) 

A. Means nothing has happened 

B. Means no measurable growth has been registered 
in the predefined growth categories 

C. Results from having both PRETEST and POSTTEST 
codes identical 

D. Could resu-l.t f rom a PRECODE of (+l) and a POSTCODE 
of (+1) 

E. Could result from a PRECODE OF (-1) and a POSTCODE 
of (-1) 

F. Could result from a PRECODE OF (O) and a POSTCODE 



19. A GAINS SCORE of (+1) 

A. Could result from a 
a PRECODE of (0; 

B. Could result from a 
a POSTCODE of (+1) 

C. Could result from a 
a POSTCODE of (0) 

D. Indicates growth 

E. Indicates decline 

F. Means the program w. 



POSTCODE of (-1) and 
PRECODE of (0) and 
PRECODE of (+1) and 

s perfect 
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20. Go back to the learning environment and examine the four questions 
on career awareness. Try to develop an individual profile which 
will identify pretest response, posttest response, posttest code, 
pretest code, and gains score for each individual, Make provision 
for a total score on career awareness. Make provision for a measure 
of decision making. 

21. After you have had a chance to look at the answer sheet giving the 
individual profile on career awareness, try to develop a group 
profile which will provide a raw count per alternative and a code 
percentage per question. This data should be open to gains scpre 
interpretation. The group profile should provide a total score 
for career awareness. The group profile should provide a total 
score for decision making. 

22. Try to repeat the above process used in question 20 in order to 
develop an individual profile for self-awt^reness based on the 

five self-awareness questions contained in the learning environment. 

23. Using the results of question 22, try to prepare a group profile 
on self-awareness. 
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ANNOTATED SUGGESTED ANSWER KEY FOR 
POSTTEST FOR 

DEVELOPING CRITERION MEASURES FOR CAREER PATH MODULES 



ANSWF .S 





- — 


1 

^ • 


n 
u 


2. 


c 




A 


A 

• 


D 
O 


J • 


iru6 


u • 


XL Uc 


7. 


True 


8. 


False 


9. 


False 


10. 


True 


11. 


True 


12. 


False 




True 


1 L 


A 

A 


1 K 


B 


16. 


C 


17. 


B, D 


18. 


B, C, D 


19. 


B, D 


20.' 




20. 


See the 


21. 


See the 


22. 


See the 


23. 


See the 



ANNOTATION 

CRT always provides criterion objectives 

NRT helps determine group percentile comparisons 

NRT provides little more than group comparisons 

and very little specific diagnosis 
NRT ranking is only as good as the level of 

competency in the group being ranked 
Individual profile 
Group profile 
Pretest response 
Pretest response 
Posttest response 
Appropriate code 
No growth situation 
All precedes are treated similarly 
Responses must be put into codes before 

treatment 
An example of a growBi situation 
An example of zero growth 
An example of growth 
An example of negative growth 
A negative score indicates the pretest 

was higher than the posttest 
A gains score of zero can result from 

many different situations 
A gains score of one indicates some 

measurable growth on predefined categories 
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CAREER . AWARENESS 
INDIVIDUAL PROFILE 

Name (Family) (First) "(Birth Date) (TEST DATE" 



GROUP IDENTIFICATION (RETEST 

DATE) 



CRITERION 
OBJECTIVE 


PRETEST 
RESPONSE 


POSTTEST 
RESPONSE 


POST- 
CODE 


PRE- 

CODE 


G^l NS 
SCORE 


SPEC 1 FY a oersona 1 
reason for going 
to school 






• 






BACK UP this personal 
reason with data 












PLAN post -secondary 
job or school in 
terms of a career 












IDENTIFY at least one 
euuca u I ona i goa I to be 
acconip 1 i shed i n 
junior high school 












CAREER AWARENESS 
(total of above four) 


X 


X 








DECISION MAKING 
(TOTAL of "0" in 
the first four codes) 


X 


X 




- 
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GROUP PROFILE 



(TEST DATE) 



- GROUP IDENTIFICATION 

CRITERION 
OBJECTIVE 




Raw Count Per 
Alternative 


(RETEST DAI 

Code-Percentage 
Per Question 


^E) 

GAINS 
SCORE 


SPECIFY a personal 
reason for going to 
school 


POST 


A 


B 


C 


D 


E 


. (+1) 


(0) 


(-1) 






















•i 




PRE 


A 


B 


C 


D 


E 


(+1) 


(0) 


(-1) 






















BACK UP this personal 
reason with data 


POST 


A 


B 


C 


D 


E 


(+1) 


(0) 


(-1) 






















PRE 


A 


B 


C 


D 


E 


(+1) 


(0) 


(-1) 




















PLAN 00<5f" — <ipnnnf1iJY*v 

job or school in terms 
of a career 


POST 


A 


B 


C 


D 


E 


(+1) 


(0) 


.(-1) 


1 
























PRE 


A 


B 


C 


D 


E 


(+1) 


(0) 


(-1) 


















IDENTIFY at least one 
educational goal to be 
accomplished in junior 
high school 


POST 
— i 


■ A 


B 


C 


D 


E 


(+1) 


(0) 


(-1) 
















! 




PRE 


A 


B 


C 


D 


E f (+1) 


(0) 


(-1) 


















CAREER AWARENESS 

(TOTAL of above four) ] 


POST 














PRE 












DECISION MAKING 

(TOTAL of "0" in 

the first four codes) 


POST 
















PRE 
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-S E L F - A W A R E N E S S 



INDIVTDUAT. PROFILE 

Name (Family) (First) (Birth Date) (l^EST D.Vii) 



GROUP IDENTiriCATION (RETEST DATE) 

I 

CRITERION 



objective j pretest 

Iresponse 


POSTTEST 
RESPONSE 


POST- 
CODE 


PRE- 
CODE 


! GAINS SCORE 


IDENTIFY at least two 
areas of interest 
IN SCHOOL 












IDENTIFY at least two 
areas of interest 
OUTSIDE SCHOOL 












oir juL»ir I at leasc cv/o 
ACADEMIC strengths 












SPECIFY at least two 
NGN- ACADEMIC strengths 












SPECIFY at least two 
areas where improvement 
is desired 












SELF-AWARENESS 
(TOTAL of above five) 




X 








DECISION MAKING 

(TOTAL of ''0" in 

the first five codes) 


X 


X 








• — ^ . — -4^ 








_ 
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S E L F - A W A R E NESS 



GROUP PROFILE . (TEST DATE) 



GROUP IDENTIFICATION (RETEST DATE) 



CRITERION 
OBJECTIVE 




Raw Count 
Per Alternative 


Code-Percentage 
Per Question 


GAINS 
SCORE 


IDENTIFY at least two area 
of interest IN SCHOOL 


POST 


A 


B 


c 


(+1) 


(0) 


(-1) 




















PRE 


A 


B 


c 


<+I) 


<o) 


(-1) 


















IDENTIFY at least two areas 
of interest OUTSIDE SCHOOL 


POST 


A 


B 


c 


(+1) 


(0) 


(-1) 


















PRE 


A 


B 


c 


(+1) 


<o) 


















SPECIFY at least two 
ACADEMIC strengths 


POST 


A 


B 


c 


(+1) 


(0) 


(-1) 


















PRE 


A • 


B 


c 


<+l) 


(0) 


(-1) 
















SPECIFY at least two 
NON-ACADEMIC strengths 


POST 


A 


B 


c 


(+1) 


(0) 


(-1) 


















PRE 


A 


B 


c 


(+1) 


(0) 


(-1) 
















SPECIFY at least two areas 
where improvement is 
desired 


POST 


A - 


B 


c 


(+1) 


(0) 


(-1) 


















PRE 


A 


B 


c 


(-rl) 


(0) 


(-1) 
















SELF-AWARENESS 
(TOTAL of above five) 


POST 


/ 


X 


X 


(+1) 


(0) 


(-1) 










PRE 


V 






(+1) 


(0) 


(-0 








DECISION MAKING 

(TOTAL of ♦^O" in 

the first five codfjs) 


POST 


•\/ 






(+1) 


(0) 


(-1) 










PRE 


' \ 


X 


X 


(+1) 


(0) 
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SUGGESTED RIRLIOGRAPHY 
RECOMMENDED PUBLICATIONS 



The toll owing MICROFICHE in the ERIC collection 
and summar.zed in RESEARCH IN EDUCATION (RIE) are called 
to the attention of educators wishing to update staff 
and learning centers : 

E RIC CODE CONTENTS 

ED 056-994 HOW TO CLARIFY INSTRUCTIONAL GOALS 

ED 074-032 HOW TO INDIVIDUALIZE CLASSROOM 
INSTRUCTIONAL GOALS 

ED 086-988 HOW TO KEEP EDUCATIONAL RESEARCH FROM 
GATHERING DUST ON A SHELF 

ED 088-861 PLANNING AND IN-SERVICE EDUCATION 

ED 085-354 CHECKLIST ON HOW TO TAKE A GOAL APART AND 
HOW TO PUT IT BACK TOGETHER IN AN 
INSTRUCTIONAL SYSTEM 

ED 090-175 ' -SEVEN STEPS TO BETTER OBJECTIVES 

ED 090-227 ROLE OF THE RESPONSIBLE LEADER IN 

DEVELOPING TEACHER-PREPARED RESEARC-H 

ED 090-976 CISNE - COMPUTERIZED INFORMATION SYSTEM 
FOR NURSING EDUCATORS 

EO EVALUATING COOPERATIVE EDUCATION 

EO LEADERSHIP BY REINFORCEMENT 

EO 092-595 AN EVALUATION SYSTEM THAT DOESN'T 
REQUIRE HYPER-EXPERTS 
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